Benchmarks, Testbeds, Controlled Experimentation, and the Design of Agent Architectures Experimentation, and the Design of Agent Architectures

The methodological underpinnings of AI are slowly changing. Benchmarks, testbeds, and controlled experimentation are becoming more common. While we are optimistic that this change can solidify the science of AI, we also recognize a set of diicult issues concerning the appropriate use of this methodology. We discuss these issues as they relate to research on agent design. We survey existing testbeds for agents, and argue for appropriate caution in their use. We end with a debate on the proper role of experimental methodology in the design and validation of planning agents.